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In many situations, there are a number of algorithms we could use to solve 
a problem, and so we need to choose which one to use. This choice could be 
based on a number of factors. One factor that might influence our decision 
is the speed of the algorithms we are trying to decide among. If algorithm 
A is faster than algorithm B, then that is one possible reason for choosing 
algorithm A over algorithm B. 

The problem comes when we try to define what “faster” means. One 
thing we could do is to run both algorithms and time them with a stopwatch. 
However, things like processor speed and compiler optimizations (i.e. how 
good the translation from high-level code to machine code was) can affect this 
result, and those things really don’t have anything to do with the abstract 
description of an algorithm. So, we’d prefer a more mathematical way of 
comparing two algorithms, so that we can focus on the key details of the 
algorithm without considering what platform things are running on, what 
compiler was used, or other such implementation details that have nothing 
to do with the inherent properties of the actual solution description itself. 

We also have to take into account data size. Knowing how fast we can 
sort an array of 10 elements isn’t particularly useful information, because we 
aren’t always going to be sorting 10 elements. On the other hand, if we can 
describe the running time as a function of a general number of elements, n, 
then that is more useful. If we increase our data size from n to nt+1, then our 
running time function will give us the new running time if we substitute n+1 
for n in the function. 

So, we will start by considering operations that do not depend on n. Mean- 
ing, some parts of a function take exactly the same amount of time whether 
n is 10 or 1000 or 1 million. For example: 


e Declaring a variable would be such an operation. Now, if we declare 
100 variables, certainly that will take longer than if we declare 1, but if 
(for example) we are about to search an array of elements using a for- 
loop, whether we are going to search 100 elements or 1 million elements, 
declaring a single variable i to use to run the for-loop iteration will take 
the same amount of time either way. 


e A basic operation such as comparison, addition or subtraction, a logical 
operation, or assignment would be such an operation For example, the 
time it takes to write one object location to a reference variable (i.e. as- 
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signment) does not vary based on how many assignments you eventually 
do. The time to do one addition doesn’t suddenly increase because you 
plan on doing 10000 additions instead of 100. 


Array access is such an operation as well. At first, you might not think 
so. It would seem that to get to cell (for example) 10, the machine first 
needs to move through cells 0 through 9. Therefore, it would seem that 
accessing a later cell takes longer than accessing an earlier one. 


But, actually, this is not the case. Arrays can given you (nearly) instant 
access to any cell because in memory the cells are arranged one after 
the other. So, it is possible to hold the starting address inside the array 
reference, and then use the calculation 


cell address = starting address + index * typesize 


to determine the starting address of the cell you want. For example, we 
said that variables of type int take up 32 bits. So, if we allocate an 
array of 10 ints, those ten cells take up ten consecutive 32-bit cells in 
memory. The first cell starts at some address in memory — the starting 
address of the array — and then the other cells are located immediately 
after it, one by one. But, if you use the calculation above to get the 
address of the first cell, you would get: 


address of ALO] (starting address of A) + 0 * 32 bits 
(starting address of A) + 0 


starting address of A 


and we just said above that the first cell was located at the start of the 
array. Likewise, 


address of A[2] (starting address of A) + 2 * 32 bits 


(starting address of A) + 64 


And, if A[0] is located at the starting address of the array and takes up 
32 bits, it would follow that at bit 33 of the array, we would see the start 
of A[1]. And since that takes up another 32 bits, it would follow that at 
bit 65 of array, we would see the start of A[2]. And that is exactly what 
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our calculation above tells us — that 64 bits after the starting address, 
A(2] begins. 


In other words, we can jump immediately to cell 2 by doing one mul- 
tiplication and one addition. And in general, we can get to any cell in 
the array via one multiplication and one addition. This is because ar- 
ray cells appear consecutively in memory, and so if you want A[i], that 
means (because we start indexing at 0) that you have to skip over i cells 
at (in this case) 32 bits each, and so one multiplication tells you how 
many total bits to skip over, and one addition of that bit total to the 
starting address tells you where the start of A[i] must therefore be in 
memory. You do not need to traverse down the cells one by one — a single 
multiplication and addition will give you your array cell each time, no 
matter how large the array is and no matter which particular index you 
are looking up. So, array cell access time does not depend on the size of 
the array in any way. 


Operations like those above are said to take constant time, because their 
running time is the same constant value regardless of the data size. 


Analyzing LinearSearch 


The process of trying to obtain a description of the running time of an al- 
gorithm in terms of the size of the data set, via careful mathematical analysis 
of that algorithm, is known as analyzing the algorithm or algorithm analysis. 
We are going to attempt to analyze the running time of LinearSearch, which 
we have reproduced below. 


// LinearSearch 


// - if key is in A, returns index where key 
// is located; if key is not in A, 

// returns -1 

// - search the array by inspecting it cell 

eA by cell until you have either finished 
I} inspecting the final cell or else found 
// the value 


public static int LinearSearch(int[] A, int key) 
nf 

int index = 0; 

boolean found = false; 

int answer = -1; 


while ((index < A.length) && (!found)) 


i: 
if (A[index] == key) 
ne 
answer = index; 
found = true; 
¥ 
index++; 
} 


return answer; 


} 


First, in the beginning of the method, we have three variable declarations, 
and each variable gets initialized as well. Declaring a single variable (setting 
aside some memory to store that variable) is a constant time operation, as no 
matter how much data we have, the time to declare one variable isn’t going 
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to change. Likewise, assignment (and hence initialization) is a constant time 

operation — writing a single value into a single variable will take the same 

amount of time regardless of how large the data set you have is. (Though, 

of course, copying a full set of data from one place to another would have a 

running time dependent on the data size. But we do not have that situation 

here; we are only writing one value of a primitive type into one variable.) 
So, the declaration and initialization lines are each constant time: 


int index = 0; // constant 
boolean found = false; // constant 
int answer = -1; // constant 


And, if you have a constant number of operations, each of which runs in 
constant time, the entire set of operations runs in constant time overall! For 
example, above, yes, each line’s running time is independent of the size of 
our data set...but since each line itself has a running time independent of the 
size of our data set, the three lines together have a running time independent 
of the size of the data set. The time to declare and initialize three variables 
is no more dependent on the size of the data set than is the time to declare 
and initialize one variable. ‘This illustrates an important principle: 


constant + constant = constant 
Next, let’s move to the code inside the if statement: 


answer = index; 
found = true; 


All we have here is two assignments — neither can depend on the size of 
the data set, so each line must run in constant time, and thus the two lines 
together must run in constant time. No matter how large the array gets, the 
total time needed to run those two lines once is not going to change. 

Next, let’s consider the if statement itself: 


if (ALindex] == key) 

4. 
answer = index; // constant 
found = true; // constant 


We have already said that the time to access an array cell is constant, so 
retrieving ALindex] is a constant-time operation. In addition, comparisons 
are constant time operations — it takes a single machine instruction to de- 
termine if two values are equal — and so comparing A[index] to key is a 
constant time operation. Thus, we can determine whether or not to run the 
code in this if statement in constant time. 

So, now here we have an interesting issue: how can we determine the 
running time of this if statement if we don’t know for sure whether the code 
inside runs or not (i.e. we don’t know if the condition is true or false)? And 
the answer here is, it doesn’t matter! If the condition is false, then it took 
us constant time to detect that and then we moved on. Total cost : constant 
time. If instead, the condition was true, then it took us constant time to 
determine that, and it will take us constant time to run the code inside the 
if statement, and so the total cost is constant+constant=constant. 

That is, even though if you were to time things with a stopwatch, the if 
statement would run faster when it was false (since it would skip over the 
interior code), we are not concerned with that kind of measurement. We are 
concerned with learning how the running time depends on the size of the data 
set. And in the case of our if statement, the running time of the statement 
as a whole does not depend on the size of the data regardless of whether the 
condition is true or false. It might be slightly slower when the condition is 
true, but it doesn’t depend on the size of our array either way. That is what 
we mean when we say “constant time”. We mean that the data set size has 
no effect on the running time of this particular part of the algorithm. 

This means that the code inside the while-loop runs in overall constant 
time, since both the if-statement and the increment of index are constant- 
time operations, and so the sum must be constant time as well. 


// overall if-statement is constant 
if (A[index] == key) 


af 
answer = index; 
found = true; 

og 

index++; // constant 


What about the while-loop condition? 
7 


(Cindex < A.length) && (!found)) 


The first condition, index < A.length, involves a comparison and the 
reading of length from the array. Presumably, the length field is stored as a 
variable somewhere — reading the length field of an array occurs frequently 
enough that a good virtual machine should keep track of array lengths as 
variables, rather than traversing the array to count the number of cells some- 
how each time the user wants to know the array length. So, we can consider 
that to be constant time, and since the comparison should be constant time, 
the first condition should be constant time. The second condition is simply 
the running of a logical operation (NOT) on a variable, and that can be done 
with a machine operations as well, so that is constant. ‘The same is true for 
the AND that ties both conditions together. So, regardless of the data size, 
the while-loop condition takes the same time to evaluate — and hence it is 
constant time. 

And finally, the return statement at the end of the while-loop is constant 
time as well. That will actually be clearer after you take CS232 (or ECE 291), 
but all you are doing there is writing the return value where it can easily be 
read by the calling method, and then returing control to the calling method. 
Both invoking a method and returning from one are constant time machine 
operations (though of course the method you invoke could take some time to 
run...but the actual invoking itself — the transfer of control from your method 
to the one you are calling — is constant time, and the transfer of control from 
it back to you — which is what we are discussing in this case — is constant 
time as well). 

So, we basically have the following: 


public static int LinearSearch(int[] A, int key) 
{ 


--- constant-time work ---- 


while (constant-time comparison) 
a! 


-- constant-time work -- 


} 


--- constant time work --- 


Note that all the individual constants could be different. ‘The time needed 
to run the declarations/initializations could be different than the time needed 
to return a value, could be different than the time needed to calculate the 
truth or falsehood of the comparison. But, none of them depend on the data 
set size, and that is what is important — not how long each of them take when 
timed on a stopwatch, but how each of their running times depends on the 
data size. And in this case, their running times don’t depend on the data size 
at all, and in that sense, the running times of the declaration/initialization 
section, the returning, the while-loop comparison, and the while-loop internal 
code are the same. That is, the order of growth is the same for all of them. 
The rate at which the running time increases as the data set size increases is 
the same for all of them — namely, that the running time doesn’t increase at 
all as the data set size increases. 

So, the final question, then, is — what is the running time of the while- 
loop itself? How long might it take to run? (We are concerned with the 
longest-possible running time for now, though next lecture we will discuss a 
few other options as well.) How many cells of the array might we have to 
look at? And the answer is: possibly each cell. If the key we are looking for 
is not in the array, then we will look at every single cell in the array, and find 
out the key is not in any of them. So, index gets initialized to 0, and we pass 
through the while-loop code A. length times, incrementing index once each 
time, until we finally exit when index == A.length. So, the actual code 
inside the while-loop, though it takes constant time when we run it once, 
doesn’t get run only once. It gets run a total of A. length times — and if we 
say that the size of the array is n, then it gets run n times. 

So, this gives us: 


public static int LinearSearch(int[] A, int key) 
¢ 


--- constant-time work ---- 
-- constant-time work, performed n times —- 


--- constant time work --- 


} 


For a total of: 


runTime = ci + c2*n + c3 


where: 
cl = declaration time 
c2 = while loop comparison and code time 
c3 = return time 


n = array size 


The interesting thing here, now, is that as n gets larger and larger, the 
c2*n term starts to be the dominant factor. Imagine assigning values to 
those constants (whatever values you happen to want — it doesn’t matter 
what values you choose, because this is just an example to illustrate the 
effects of the middle term, and those effects will be the same no matter what 
the individual constants are). 


runTime = 12 + 2*n + 16 


1: runTime = 12 + 2 + 16 = 30 

10: runTime = 12 + 20 + 16 = 48 

100: runTime = 12 + 200 + 16 = 228 

1000: runTime = 12 + 2000 + 16 = 2028 
10000: runTime = 12 + 20000 + 16 = 20028 
CLG. a. 
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When n is very very small, the constants cl and c3 still matter, but as 
n gets larger, their effect is negligible. There is a difference between 2 and 
30, but much less of a difference (proportionally) between 200 and 228, and 
still less of a difference between 20000 and 20028. When n is anything but 
an extrememly small, trivial value, the running time of the LinearSearch 
method can basically be expressed as a constant times n — any extra constant 
terms have only a trivial effect on the overall running time. 

Or in other words, 


runTime(LinearSearch) =c *n 
where c is a constant 
and n is the size of the array 


That is, the running time is linear. That is the key idea we are trying to 
express. The actual time in milliseconds doesn’t matter. What matters is the 
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order of growth, or proportionality, of the running time of the method. And 
here, the order of growth of the running time of LinearSearch is, well, linear! 
If you double the size of your data set (i-e., your array), then the running 
time of your method will about double. If you quadruple the running time 
of your data set, then your running time will about quadruple. The running 
time of your method is proportional to the size of the data set (proportional 
meaning, it is a constant times the size of the data set) 

This information is useful because we can use it to learn what kind of effect 
increasing our data set is likely to have. If it is taking me about an hour to 
search an array for an element (it’s a big array), then I know trying to search 
an array about twice the size should take me about two hours. ‘Trying to 
search an array four times the size should take me about four hours. But 
this has nothing to do with the operating system I am using, it has nothing 
to do with the processor in my machine, and it has nothing to do with the 
compiler I used. The fact that the running time is linear in the size of the 
array is a fundamental mathematical property of the algorithml!!. 

That means our analysis is completely platform-and-software-tool inde- 
pendent. That is the best way to compare algorithms, because it helps us 
choose between algorithms when we are still in the process of designing our 
program, and because the decisions made based on such a comparison don’t 
change when a new processor comes along because the result of the compari- 
son of two algorithms using mathematical analysis is completely independent 
of the processor you used. The algorithm that is mathematically faster today 
will still be mathematically faster next year when processors have doubled in 
speed. 
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Analyzing SelectionSort 


Before we analyze SelectionSort, we will find it helpful to put all the 
code in one place. So, instead of having SelectionSort call FindMinimum 
and Swap, we’ll move that code from those methods into SelectionSort. We 
could certainly analyze it in its previous form as well, but you are just starting 
this idea, and so for now we will make it easier for you by having all the code 
in one place. 


public static void SelectionSort(int[] A) 


{ 
int 1, j, min, temp; 
// find minimum code 
for (i=0; i<A.length; i++) 
{ 
min = i; 
for (j = i+1; j < A.length; j++) 
if (A[j] < Almin]) 
min = j; 
// swap code 
temp = ALi]; 
Afi] = Almin]; 
Almin] = temp; 
} 
Es 


We can start out the same way we did before — by trying to first identify the 
constant-time parts of the algorithm. Certainly, we’ve seen that an individual 
declaration is constant time, and so a small group of them, such as: 


int 1, j, min, temp; 


should be constant time. Likewise, the swap code simply has one or two 
array accesses per line and an assignment per line. Both array accesses and 
assignments are constant-time operations, and so each line should run in 
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constant time...and thus the entire collection of the three swap lines should 
run in constant time. 


public static void SelectionSort(int[] A) 


f 
--- constant-time work --- 
// find minimum code 
for (i=0; i<A.length; i++) 
a 
min = 1; 
for (j = i+1; j < A.length; j++) 
if (A[j] < Almin]) 
min = j; 
// swap code 
---- constant-time work ---- 
a; 
} 


Likewise, setting min to i just before the start of the inner for-loop is 
a constant-time assignment, and setting it to j within the if statement is 
constant as well. The if statement inside the inner for-loop has a condition 
that requires two array accesses and a comparison. Each array access is con- 
stant, and the comparison is constant, so determining the truth or falsehood 
of the condition should be constant. 
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public static void SelectionSort(int[] A) 
4. 

--- constant-time work --- 

for (i=0; i<A.length; i++) 


A 
--- constant-time work --- 
for (j = i+1; j < A.length; j++) 
if (--- constant-time work) 
--- constant-time work --- 
---- constant-time work ---- 
} 


As 


And, as we saw with LinearSearch, whether the if statement’s condition 


is true or false, the if statement is still constant. Sometimes, you calculate 
the condition in constant time and then do nothing. Sometimes you calculate 
the condition in constant time and then run the if statement code in constant 
time. But, the time needed to run through the if statement once does not 
depend on the size of the array regardless of whether the condition is true or 


false. 


So, we have: 


public static void SelectionSort(int[] A) 
{ 

--- constant-time work --- 

for (i=0; i<A.length; i++) 


{ 
--- constant-time work --- 
for (j = i+1; j < A.length; j++) 
- constant-time work 
---- constant-time work ---- 
} 
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Now, what about the interior for-loop? We traverse from i+1 to A.length-1 
there, so whatever we are doing inside this loop, we are doing it A.length - 1 - i 
times. And the code inside the loop runs in constant time. So, the inner loop 
can be thought of as being proportional ton - 1 - i. 

So, the code inside the outer for loop can then be thought of as: 


runTime of one pass through outer loop = 
ed # ini) 4c2 3 


where ci = time to run min=i 
c2 = for-loop control and if statement, one time through 
c3 = time for if statement 


Overall, this will depend on n and i — in particular, it is linear in terms of 
n-i. As we saw last time, the constants don’t tend to matter — so we can 
discard c1 and c3. And, the “-1” term in n-1-i doesn’t really matter either. 
But both n and i can vary, and so the running time through one pass of the 
outer loop depends on both of them. 

So, that means we have: 


public static void SelectionSort(int[] A) 


{ 
--- constant-time work --- 
for (i=0; i<A.length; i++) 
{ 
-- work that takes time proprotional to n-i 
} 
} 


We’d prefer to just express things in terms of n. So, how can we calulcate 
the overall running time of this method, taking into account the fact that i 
changes, and so each time through the interior for-loop will be a bit faster. 
Well, we can just write things out: 
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time #x through running time 


the outer loop i of inner loop 
1 n= 
1 n-2 
3 n-3 
ined n-2 1 
n n-1 Ox 
*it won’t actually take ‘‘0’’ time -- remember, there 


are constants we left out of our analysis 


And the running time should then be the sum of all the values in the 
right-hand column, i.e. the overall running time will be the running time of 
the outer loop, which will be the sum of the running times of all the different 
passes through the inner loop. 


ON AAD Sai tot ie te “eee 


runTime == ------ eas 
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1: runTime .5 - .5 = Ox 
10: runTime 50 - 5 = 45 
100: runTime = 5000 - 50 = 4950 
1000: runTime = 500,000 - 500 = 499,500 
10000: runTime = 50,000,000 - 5000 
= 49,995,000 


B Bp BB B 
ll 


*again, the algorithm wouldn’t really run in 
‘““zero time’’ -- there are constants we left out 


Or, what really controls the value of the final result here is the “one-half 
times n squared” term, and not the “one-half times n” term. That is, in the 
same way that small constants were trivial in a linear function, linear terms 
are trivial in a quadratic function. As n gets larger, the highest order term 
quickly starts to dominate the end result, so the highest order term is the 
only one you really need to focus on. 

So, the running time of SelectionSort is quadratic — meaning, the running 
time growth is proportional to the square of the data set size, instead of just 
being proportional to the data set size itself as linear algorithms were. If you 
double the data set size, you quadruple the running time. If you have a data 
set three times as large, SelectionSort will take nine times as long. So, 
you can see that the running times for quadratic algorithms get much larger, 
much faster than the running times for linear algorithms do. (You can graph 
the functions y = z and y = 2? to see this for yourself.) This means that 
a linear-time method will (except possibly for very very small arrays), run 
faster than a quadratic-time method on the same data set. For example, we 
will be able to search an n-element array using LinearSearch much faster 
than we will be able to sort it using SelectionSort. 
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Today, we are going to begin by analyzing the performance of InsertionSort 


public static void InsertionSort(int[] A) 


{ 
oC a een Bore 
for (i = 1; i < A.length; i++) 
1 
int temp = ALi]; 
0 ae ae 
while ((j>=0) && (AL[j] > temp)) 
{ 
A(j+i1] = ACj]; 
Jers 
} 
A[j+i] = temp; 
} 
A; 


Based on our discussion of constant-time during the last lecture, it should 
be clear that the following lines are all constant time: 


int i, j; // merely declarations 


int temp = ALi]; // array access and assignment 
Jie 213 // subtraction and assignment 


AC{jt+ti] = ALj]; // 2 array accesses, one 
// addition, one assignment 


cle // subtraction 


A[j+1] = temp; // one addition, one array 
// access, one assignment 
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In addition, it should take only constant time to evaluate the while-loop 
condition: 


(j>=0) // one comparison --- constant 
(ALj] > temp) // an array access and a 

// comparison --- constant 
&& // logical operator -- constant 


Hence, ((j>=0) && (ALj] > temp)) is constant 
This gives us: 


public static void InsertionSort(int[] A) 


a4 
--- constant-time work --- 
for (i = 1; i < A.length; i++) 
aa 
--- constant-time work --- 
while (--- constant-time work ---) 
- 
--- constant-time work --- 
} 
--- constant-time work --- 
} 
a 


So, how much times does the code inside the while loop run? Well, j, which 
starts at i-1 could become as low as -1 before we exit the while-loop via 
the first condition. So, it is possible that i decrememts of the value j occur, 
and thus it is possible that the while-loop runs in time proprotional to i (i.e. 
“linear in i”). 

So, the code interior to the while loop runs i times each time we run the 
while-loop itself. The problem is that i is different each time we run the 
while-loop. So, if we want to know how many times the code inside the for- 
loop runs — which is essentially what we are asking when we ask how long 
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InsertionSort takes — we need to find the sum of the different while-loop 
running times, just as we did for SelectionSort. 


Pass #X through running time of 
for loop 1 while loop 
al 1 1 
2 
3 3 3 
n= 1 no n-1 


So, just as before, 


dD 0S: do a ED Ae, SSeS 


and proceeding from this point using the same analysis we used for SelectionSort, 
we arrive at the conclusion that InsertionSort is quadratic, just as SelectionSort 
is. 

Up to this point, the analysis we have done is what is known as worst-case 

analysis. In this kind of analysis, we are concerned with finding out how 
long the algorithm takes to run in the worst possible situation. For example, 
you are designing software to pipe extra oxygen into the space shuttle living 
area for our astronauts in the event that their existing oxygen all leaks out 
somehow, you’d want to make sure that the algorithm you chose to make 
this oxygen delivery system work was very very fast, even in the worst case. 
It would do you no good if it “usually” worked quickly, but “hit a snag” in 
certain situations and spent an extra hour processing before piping in oxygen. 
The astronauts would die waiting for your algorithm to finish!! So, in such a 
case it would be important to know the absolute longest that your algorithm 
could take, no matter what input the algorithm is given. That is the idea of 
worst-case. 

On the other hand, often what we are more concerned with is what usually 
occurs. We might not like using a computer system that took 10 minutes to 
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send a file to the printer each time we wanted to print. But, if it usually 
printed right away, and only took 10 minutes every once in a long while due 
to some quirk in certain types of files, then we might find that acceptable. 
In fact, we might prefer that to a situation where the worst case was only 
5 minutes, but that was what “usually” happened as well. Sometimes we 
will prefer to focus on what “usually” happens, and pick an algorithm that 
“usually” runs very fast, even if occasionally we end up with a slower case as 
a result. 

This notion of “usual” is known as the average case, and attempts to 
analyze an algorithm with the average case in mind are known as average 
case analysis (in contrast to worst case analysis). 


e LinearSearch is also linear in the average case. 
e SelectionSort is also quadratic in the average case. 


e InsertionSort is also quadratic in the average case. 


InsertionSort in particular, though has a neat property — linear in the 
“best” case. That is, when arrays are already nearly sorted, InsertionSort 
works in linear time, because you are never really shifting any one element 
to the left more than a few cells. The more unsorted it is, the “closer to 
quadratic” we get. 

But generally, we don’t bother with “best case” analysis, because “best 
case” is usually only a very specific situation. For example, best case for 
LinearSearch is when we find our value very very early in the array. We 
only bring up “best case” for InsertionSort because in the real world, data 
is often partially sorted already, and some algorithms can also leave data in 
a partially sorted state, and so the idea that InsertionSort can complete 
a partially-done sort in linear time is a useful fact to be aware of in many 
situations. 
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